EULER CHARACTERS AND SUPER JACOBI 
POLYNOMIALS 



A.N. SERGEEV AND A.P. VESELOV 

Abstract. We prove that Euler supercharacters for orthosymplectic 
Lie superalgebras can be obtained as a certain specialization of super 
Jacobi polynomials. A new version of Weyl type formula for super Schur 
functions and specialized super Jacobi polynomials play a key role in the 
proof. 
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1. Introduction 

The main purpose of this paper is to develop further the link between the 
theory of the deformed Calogero-Moser systems and representation theory 
of Lie superalgebras [HE]. Recall that in the BC{m,n) case the deformed 
Calogero-Moser systems depend on 3 parameters k,p,q. For generic values 
of these parameters they have polynomial eigenfunctions SJx{u,v;k,p,q) 
called super Jacobi polynomials [3]. The special case k = p = —l,q = 
corresponds to the orthosymplectic Lie superalgebra osp(2m + 1, 2n). 

It turns out that this case is singular in the sense that the corresponding 
limit does not always exist. However if we consider first the limit when 
k — > —1 with generic p,q and then let {p,q) (—1,0) then this limit does 
always exist and gives what we call specialized super Jacobi polynomials 
SJx{u,v] —1, —1,0). A natural question is what do they correspond to in 
the representation theory of orthosymplectic Lie superalgebras. We show 



that the answer is given by the so-cahed Euler characters studied by Penkov 
and Serganova [U [5] . 

There is a classical construction due to Borel, Weil and Bott of the ir- 
reducible representations of the complex semisimple Lie groups G in terms 
of the cohomology of the holomorphic line bundles over the corresponding 
flag varieties G/B (see e.g. [6], section 23.3). Such line bundles La are de- 
termined by the weight A G f)*, where f) is Cartan subalgebra of Lie algebra 
of G and B is Borel subgroup of G. The cohomology groups H^{G/ B, Lx) 
are finite-dimensional and have natural actions of G on them. By Kodaira 
vanishing theorem all of them are zero except one depending on the Weyl 
chamber the weight A belongs to (see details in [H [7]). In particular, for a 
dominant weight A the space of sections H^{G/ B, Lx) gives the irreducible 
representation with highest weight A. 

In the Lie supergroup case in general there is no vanishing property, so 
this construction does not work [4j. The idea is to consider the virtual 
representation given by the Euler characteristic 

£x = Y.{-iyH\G/B,Ox) 

i 

for certain sheaf cohomology groups (see [5j). For the generic (typical) 
highest weights A this leads to the Kac character formula [8]. 

One can generalise this construction for any parabolic subgroup P of G 
in a natural way. In the orthosymplectic case G = 0SP{2m + 1, 2n) there 
is a natural choice of P with the reductive part GL{m,n). The correspond- 
ing supercharacter Ex (called Euler supercharacter) can be given by the 
general explicit formula due to Serganova [5]. Our main result is that Ex 
up to a constant factor coincides with the specialized super Jacobi polyno- 
mials SJx{u,v; —1, —1,0). A similar result holds for the Lie superalgebra 
osp(2m,2n) and super Jacobi polynomials SJx{u,v; —1,0,0). 

The proof is based on a new formula for super Schur polynomials and the 
super version [3] of Okounkov's formula for Jacobi polynomials [9]. We prove 
also the Fieri and Jacobi-Trudy formulas for the corresponding specialized 
super Jacobi polynomials and Euler supercharacters. 

It turns out that we can simplify the relations with super Jacobi poly- 
nomials if we choose a different Borel subalgebra and the corresponding 
parabolic subalgebras following recent work by Gruson and Serganova [T0|. 
In that case the Euler supercharaters coincide with specialised super Jacobi 
polynomials without non-trivial factor (see the last section for the details). 

This shows that the super Jacobi polynomials can be considered as a nat- 
ural deformation of the Euler supercharacters and gives one more evidence of 
a close relationship between quantum integrable systems and representation 
theory. 
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2. Weyl type formulas for super Schur polynomials 



We start with the new formula for super Schur polynomials, which will 
play an important role in this work. 

Let H{m, n) be the set of partitions with Am+i < n, which means that 
the corresponding Young diagram belongs to the fat {m,n)-hook. Let A be 
such a partition and let d = m — n be the superdimension. Introduce the 
following quantities 

i(A) = max{i \ Xi + d — i > 0, 1 < i < m}, (1) 
j(A) = max{i j a;. - d - i > 0, l<j< n}. (2) 
If all Aj + d — i < then by definition i(A) = (and similarly for j(A)). It 
is easy to verify that in all cases m — i{X) = n — j{X), so i(A) — j(A) = d. 

We should mention that Moens and van der Jeugt introduced a similar 
quantity A;(A), which they called (m, n)-index of A (see Definition 2.2 in 
[19J). They were motivated by Kac-Wakimoto formula [11]. It is related to 
our i{X) by i(A) = A;(A) — 1. Moens and van der Jeugt used this quantity 
to write down a new determinantal formula for super Schur polynomials 
different from Sergeev-Pragacz formula |12j . 

Our formula ([7]) below is another new formula of Weyl type, which gen- 
eralizes Sergeev-Pragacz formula. Let us denote by vr;^ the set of pairs 
such that i < i{X) or j < j(A) and fix a partition i/ such that 

A n Um,n ^ ^ tta, 

where Il^.n is the rectangle of the size m x n. When u = ttx the formula 
([7|) coincides with Serganova's formula (jlljl for a special choice of parabolic 
subalgebra depending on the weight (although that was not the way we 
came to this). 

Introduce the following quantities by 

k = Xi + m — Ui — i, 1 < i < i{X), li = m — i, i{X) < i < m, (3) 

kj = X'j+n- u'j -j,l<j< j(A), kj=n- j, j{X) < j < n. (4) 
Now we can formulate the main result of this section. Recall that super 
Schur polynomial is the supercharacter of the polynomial representation 
M = of Ql{m, n) determined by a Young diagram A from the fat hook 
H{m,n). It can be given by the following Jacobi-Trudy formula (see [13]): 

^Ai ^Ai+l • • • ^Ai+«-l 

h\2-l ••• /lA2+«-2 



SPx{x,y) 



(5) 



where / = Z(A) is the number of non-zero parts in the partition A, /i^ = 
hk{x,y) are determined by 



and ha = if a < 0. One can check that the highest coefficient of SPx{x, y) 
is equal to (—1)^, where b = ^j- This explains the appearance of this 

sign below (see also [H 



Theorem 2.1. The super Schur polynomial SPx{xi, . . . , Xm, Hi, ■ ■ ■ , Un) can 

be expressed by the following Weyl type formula for any choice of partition 
V such that A fl Il^.n C i/ C tta : 

SP\{xi,. . . ,Xm,yi, ... ,yn) = 



n ~ 



■^1 • • • -^m tfl ' ' ' Un 

A{x)A{y) 



(7) 



where A(x) = UT<ji^^ " ^^i)' ^iv) = ]Xl<M " ^i)' ^ = Ej>m ^i- 

Proof. For any function f{x,y) define the following alternation operations 

{f{x,y)}= E e{w)w{f{x,y)) 



WGSmXSn 



and 

{/(a:,y)}x= ^ e{w)w{f{x,y)), 
weSm 

where Sm and Sn permute Xi and yj respectively. Introduce also the nota- 
tions 

™Pm _ „m-l m-2 p„ _ n-1 n-2 

X — X^ X2 . . . Xj^, y — i/i y2 ■ ■ ■ Un- 

First let us prove the following equality 



{ha{x,y)xP^yP-} ={\{{xi- yj)xi 
.i=i 



a+d~l m,~2 „ n—l 



X 



...xlyr'...y'n}, (8) 



where a is an integer such that a + d—1 > 0. Indeed, we have from the usual 
Weyl formula for a > 

X^ayX )X^ X2 ■ . . ■J'^rnJx — l*'^! "^2 ' ' ' mJx 

This is true also for all a > 1 — m because the left hand side for negative a 
is zero by definition. From ([6]) we have 

n 

hk{x,y) = ^{-iyhk-j{x)ej{y), 

j=0 

where h^ and ej are complete symmetric and elementary symmetric poly- 
nomials respectively. Note now that ifa + d— 1>0 and < j < n we have 
a — j > 1 — m. Therefore in that case 

n 

{ha{x,y)xr' ■ --x'^h = Y.{ha-,{x)x^~' . ..xlU-iye,{y) = 

j=0 
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j=0 



j=0 



. m-2 \ 
•^2 ■ ■ ■ •^mSx- 



This implies the formula ([H]). 

We prove now the theorem by induction. For this we need the following 

Lemma 2.2. If \i + d — 1 > ^ we have the following equality 



{SPx[x,y)xP^yP-} = \ J{{xi-yj)x\'+'''^SP^^{x,y)x 



m-2 p 

■ ■ ■ ■^mil 



where x = {x2, . . . , Xm), A = (A2, A3, . . . ). 



(9) 



Proof. To prove it we use the Jacobi-Trudy formula ([5]). Let Ai + d — 1 > 0, 
then we have 



{SPx{xi,...,Xm,yi,...,yn)x'^ ^...x^y" . . . y^} 



h\i ^Ai+i 
hx2~-i 



■ ■ h.X2+l-2 

hxi-i+i hxi-i+2 ••• hxi 

{hx,x^-'...y^^} {/.A.+ixr'---y^} 
hx2-i hx2 



m— 1 „ n—1 







"1 



Vn ^1 • ■ - Vn 



■^mVl ■ ■ ■ Vn 



{hx,+i-ixr'...y^^} 
h\2+i-2 

K 

Xi+d+l-2 „,0 
■ • • ''^1 



h 



■A2-I 



h 



hxi-i+i 



hx2+i- 



Vn 



hxi-l+2 

Now multiplying every column except the last one by xi and subtracting it 
from the next column taking into account the equality 

ha-i{x, y) - xiha{x, y) = ha{x, y) 

we get 

h\2 ^A2+l • • • hx2+i~2 



-1+2 



-1+3 



h 



A; 



where hf^ = hk{x,y) and x = {x2, ■ ■ ■ ,Xm) as before. This implies the 
formula Q. □ 



Lemma 2.3. Let Ai < n, then we have the following equality 

{SPx{x, yi,.. .,yn)xP"^y^"} = {SPx{x,yi, . . . , . 
Proof. We have (see formulas (5.9) in [18] and (25) in [14J) 

SPx{x,y) = j;(-l)l^l5A/^(x)5^,(y) 

So, if Ai < n then fi'„ = and lemma follows. □ 

Now we finish the proof by induction in m + n. Note first that the case 
n = follows from the classical Weyl's formula, while in the case m = we 
have also the sign (_i)-*'m+i+Am+2+ -- determined as in the formula (25) in 
[14j . Let us assume now that m > 0. Rewrite formula ([7]) in the form 

{SPx{x,y)xP-yP"} = {-l)'^} H {x, - y,)x[K . . x^y^,' . . . y^^A . (10) 

If z^i < n (and hence Ai < n) by induction assumption we have 
{SPx{x,yi, . . . ,yn-i)xP"^y^''-'} = 

(-l)'l n {x,-y,)x[K..xty'^^-\..ytT' 

Multiplying both sides by the product yi . . . y„ and using lemma 12.3^ we 
prove the theorem in this case. 

If vi = n then the box (l,ra) belongs to ttx and therefore by definition 
either i{X) > or j(A) = n. If i{X) = then j(A) = n — m < n since m > 0. 
Thus, i{\) > 1, which implies that Ai + d — 1 > 0. This means that we can 
apply lemma [2^21 By induction we have 

{SPx{x2,...,Xm)x^-\..xlyP-} = 



where z) is the partition i' without the first row. Now the theorem follows 
from lemma [2^ □ 



3. EULER SUPERCHARACTERS FOR LlE SUPERALGEBRA 05p(2m + l,2n) 

In this section we consider the case of orthosymplectic Lie superalgebra 
osp(2m + 1, 2n), the case of osp(2m, 2n) is considered in section 7. 

Recall the description of the root system of Lie superalgebra g = osp(2m+ 
l,2?i). We have g = go©gi, where go = so{2m+l)(Bsp{2n) andgi = Vi^V2 
where Vi and V2 are the identical representations of so(2m + 1) and sp{2n) 
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respectively. Let ±ei, . . . , ie^, ±5i, . . . , ±5„ be the non-zero weights of the 
identical representation of g. The root system of osp{2m + 1, 2n) consists of 

Ro = {±ei±£j,±ei, ±6p±5q,±25p, i ^ j, I < i,j < m ,p ^ q, 1 < p,q < n}, 

Ri = {±£i ± Sp, ±6p}, Riso = {iej ± Sp}, 
where Ro,Ri and Riso are even, odd and isotropic parts respectively. The 
invariant bilinear form is given by 

(ei, Ej) = 1, {ei,ej) = 0, i ^ j, {6p, 6p) = -1, {6p, 6g) = 0, p q, {ei,6p) = 0. 

The Weyl group Wq = {Sm x Z™) x {Sn x ^2) ^^ts on the weights by sep- 
arately permuting £i, j = 1, . . . ,m and 5p, p = 1, . . . ,n and changing their 
signs. A distinguished system of simple roots can be chosen as 

B = {Si — 62, ■ ■ ■ , Sn-l — 6m Sn — — £2, • • • , Em-l — ^rn, ^m}- 

Introduce the variables Xj = e^*, xj^^ = e^*/^, Ui = Xi + x~^, i = I, . . . ,m 
and yp = e^p, Vp = yp + y~'^, p = 1, . . . ,n. 

For any parabolic subalgebra p C and any finite-dimensional represen- 
tation M of p by a superversion of Borel-Weil-Bott construction one can 
define the corresponding Euler supercharacter E^(M). According to the 
general formula due to Serganova [5] 

X ^ . DePschM \ 




with 

Here p is the half-sum of the even positive roots minus the half-sum of odd 
positive roots, Rp is the set of roots a such that g±a C p (see formula (3.1) 
in [5]). Note that we use here the supercharacter rather than the character 
used by Serganova. This leads simply to the change of signs in some places. 
Consider now the parabolic subalgebra p with 

Rp = {^i - £j, 5p — 6q, ±{ei - 6p)}. 

The algebra p is isomorphic to the sum of the Lie superalgebra Ql{m, n) and 
some nilpotent Lie superalgebra. In that case Serganova's formula ([IT]) has 
the form 

( n (i-^^"V)^r^---^iyr^---yi^c/iM^ 



E{M) = ^ w 



11 11 



A(^)A(^;)nili(4 -xr^)n-=i(yj +2/7^) 

V / 

(12) 

where Ti^^n is the rectangle of the size m x n, A(u) = ni<j(^j ~ ^i)' A(w) = 
W^^jivi — Vj) and Ui = Xi + x~^,Vj = yj + yj^ as before. 



The following proposition explains the appearance of the powers of 2 in 
the later considerations. As far as we know the appearance of a power of 
2 coefficient was first noticed by Cheng and Wang [T7]. It should have a 
geometrical explanation but we give here a direct algebraic proof. 

Proposition 3.1. For the trivial even representation M we have 

^(M) = 2™'^('"'"). (13) 

Proof. It is well-known (see e.g. [18j, Ch. 1, formula (4.3')) that 

- = ^(-l)l^l5A(xr\ . . . , X-l)5A'(yr\ • • • , Vn'), 

id 

where Sx is the Schur polynomial and the sum is over all the Young diagrams 
A, which are contained in the (m x n) rectangle, A' is the diagram transposed 
to A. Using the Weyl formula for Schur polynomials we can replace in the 
formula (fT2]) the product Ilj j(l ~ ^T^yJ^) 



One can check that the only non-vanishing terms correspond to the symmet- 
ric Young diagrams A = A'. Now the proposition follows from the fact that 
the number of the symmetric diagrams contained in the (m x n) rectangle 
is equal to 2™™^™'"'), which can be easily proved by induction. □ 

Consider now the polynomial representations M = of 0l(m,n) deter- 
mined by a Young diagram A € H{m, n) with the supercharacter given by 
the super Schur polynomial: 

SchVx = SPx{xi,...,Xm,yi,---,yn)- 

The next step is to rewrite the formula for the Euler supercharacter 

Ex = E{V^), (14) 

using the formula ([7]) for the super Schur polynomials in the case when 
u = TTx- In this CcLSG Ij are defined by 

k = Xi + d — i, I < i < i(A), li = m — i, i{X) < i < m, (15) 

kj=X'j-d-j,l<j<j{X), kj = n-j,j{X)<j<n, (16) 

where d = m — n. 

Introduce the following polynomials (which are particular cases of classical 
Jacobi polynomials, see the next section) 

Mz) = Yj^ TTT^' Mz) = ^ _i/2 ' (l*") 

where z = w + w^^. Define also Hxi^, v) as 

Ux{u,v) = Yi (ui-Vj). 



Theorem 3.2. The Euler supercharacters can be given by the following 
formula 

E\{u,v) = 



C(A) ^ 

Ulg5m X5n 



Tl\{u,v) 



ifl^ (m) . ■ ■ ifl^{Um)lpk^ {Vl)--- ^kAVn) 

A(n)A(?;) 



(18) 



where C{\) = (-1)^2™-^^ = (-1)''2"-JW, b = Ej>m>^j andi{X), j(A), h, kj 
are defined by {2P, ^M>,^IM>, ^IM>■ 

Proof. According to Theorem 12. II for M = the corresponding superchar- 
acter schM = SP\ can be given by ([7]), or, equivalently, by ([TO]) . Substitut- 
ing this into Serganova's formula ()12p we get 



E{M) = {-If Yl ^ 



(*j)en 



\ 



J 



where we use the notations 



X 2 = X 



xm , y =yi ■■■yn 



Since 



n (l-xriy7i) J] {xi-y,)= W {ui-vj)=lix{u,v), 

(ij)G7rA {i,j)e-Kx {i,j)eTTx 

we have 



E{M) = {-If ^ 
weWo 



(i,j)eUm,n\TTx 



A{u)A{v)UT=iix! -x;nUUiyf +yjn 



Now proposition 13.11 and Weyl's formula for the BCn root system (which is 
the root system of the Lie superalgebra osp(l, 2n)) allow us to replace here 
n(.,,)en^.„w(l - ^r'y7') by = 2«-^(^) to come to 

/ \ 



E{M) = C(A) Y ^ 

uieWo 



Il\{u,v)x '^ 2y 



A{u)A{v)\{{xJ -x^ ~n\{{y]+y]~n 

i=i j=i 



■ (19) 



Summing now over the subgroup x C Wq and using (jl7p we have the 
claim. □ 

We win use this formula now to show the relation with super Jacobi 
polynomials. 
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4. Super Jacobi polynomials for k = —1 



The main result of this section is the fohowing Weyl-type formula for the 
super Jacobi polynomials [3] with k = —1. Let us introduce the following 
polynomials fi{z,p,q), which are certain normalised versions of the classical 
Jacobi polynomials with a = —p — q — \i (i = q — \'- 



Mz,p,q)=Y,Ci4z-2y, 



j=0 

where Ci^i = 1 and 

+ {1-1)1 {i + l-p-q-l/2)...{l-p-q-l/2) 

{l-iV- {l + i-p-2q)...{2l-l-p-2q) 

for i < /. Introduce also 

9k{w,p,q) = fk{w,-p,-l - q). 

Note that the polynomials (f{z),ilj{z) from the previous section are the par- 
ticular cases: 

=/i(z,-l,0), ipk(z) =gk(z,-l,0) = fk(z, 1,-1). 

Later we drop the parameters for brevity, writing simply fk{z),gk{z)- 

Let A E H{m,n) be a partition from the fat hook and tta, h,kj be the 
same as in the previous section (see formulas ()15p . ()16p above). 

Theorem 4.1. The super Jacobi polynomials for special value of parameter 
k = —1 can be given by the following formula 



SJxiu,v,-l,p, q) 

WGSmXSn 



^^^^'^^ fl,{ui) . . . fl^{Um)gki{vi) ■ ■■gkAVn) 



(20) 



_A(ii)A(i; 

where as before b = Y.j>m^j ^'^'^ ^x{u,v) = n(jj)G7r;,(^i " ^i)- 

We prove this first in the particular case when A contains the m x n 
rectangle, i.e. 

The corresponding formula for super Jacobi polynomials can be considered 
as a natural analogue of Berele-Regev factorisation formula for super Schur 
polynomials [15j. For such a diagram A one can consider its sub-diagram /i, 
which is the diagram A without first n columns. Define 

Wi{X) = Hi, i = l,...,m, Zj{X) = X'j, j = l,...,n. (21) 

In other words, Wi is the length of i-th row of /i and Zj is the length of j-th 
column of A G Hm,n- 

10 



The super Jacobi polynomials [3j in the special case k = —1 can be defined 
for generic p, q in terms of super Schur polynomials SP\ by Okounkov's 
formula 

SJx{u,v,-l,p,q) =Y,K^-^SP-^{u,v) (22) 
ACA 

where u = {ui, . . .,Um), v = {vi, . . . u = {ui - 2, . . . ,Um - 2), v = 
{vi - 2, . . . ,Vn - 2) and 

^ _ .,A|-|A| C'xid) Clid -p-q-\) h{w{X),z{X), -1, h) 
C?(d)C7?(d-p-,-i) C,-(l)C+(2/.-l) • 

Here d = m — n is the superdimension, h = d — ■^p — q, 

^a(^)= n (A. + j-(A;.+i)+x), (24) 

Cxi^)= n (Ai-J + (A;-i)+x), (25) 
fo-)eA 

Ca°(x)= n {j-l-{i-l) + x), (26) 

fe-)GA 

Ix{w, z, —1, h) is the specialisation of the deformed interpolation BC poly- 
nomial I\{w, z, k, h) (see Proposition 6.3 in [3j), w{X) and z{X) are defined 
by ()2ip above. We should note that we are using here more convenient 
variables 

Ui = Xi + Vj = Vj + yj^, 

rather than Uj = \{xi + x^^ — 2),Vj = ^{yj + y~^ — 2) used in [3]. 

Theorem 4.2. Let X € H{m, n) contains the m x n rectangle. Then the 
super Jacobi polynomials SJx{u,v, —l,p,q) can be expressed in terms of the 
usual Jacobi polynomials as 

m n 

SJx{u, V, -l,p, q) = (-l)l''l W Jl('Ui-fj) J^(n, q)Jy{v, -1, -p, -l-q), 

1=1 j=i 

(27) 

where 

Hi = Xi - n, i = 1,. . . ,m, Uj = X'j - m, j = 1, . . . ,n. (28) 

Proof. To prove this we need the following factorisation formula for the 
deformed interpolation BC polynomials. We should mention that in the 
special case k = —\ these polynomials are the particular case of the factorial 
super Schur functions considered by Molev in [16j , but in our particular case 
one can give a simple direct proof. 

11 



Lemma 4.3. If X H{m, n) contains the m x n rectangle, then we have 
the following formula for deformed interpolation BC polynomials 

m n 

h{w, z, -1, h) = i-lt^ Yl H [{wi +n + h-if -{zj-h-j + if] X 

i=ij=i 

Ifiiwi, . . .,Wm, -1, h + n)Iv{zi - m, . . . ,Zn - m, -1, 1 - h + m), 

where I^i^w, k, h) is the usual interpolation BC polynomial and /u, u are the 
same as in the theorem. 

Proof. Let us denote by I\ the right hand side of the previous equahty. 
According to proposition 6.3 from [3] it is enough to prove that I\ satisfy 
the fohowing properties: 

1) I\ is a polynomial in variables {wi + h + n — if, i = 1, . . . ,m and 
{zj -h + l-jf, j = l,...,n; 

2) the degree of this polynomial is 2[A|; 

3) ixiwCX),z{~X)) = if A € H{m,n) and A ^ A; 

4) h{w{X), z{X)) = Ui^,J)ex(^ + A. - j + A;. - i){2h _ i + A, + j - A^- - i). 

First two statements are obvious from the explicit form of Ix . The fourth 
statement can be checked directly. Let us prove the third property. 

Let us suppose that A ^ A. Consider two possible cases depending on 
whether A contains the m x n rectangle or not. In the first case we have 
H ^ jl or u <^ u . Therefore by definition of the interpolation BC polynomials 

m 

Ifj,{il, -1, h + n)Iv{v, -1, 1 - /i + m) = 0. 

In the second case consider the box {i,n), 1 < i < m such that (i,n) ^ A, 
but (z — l,n) € A (if i=l, we require only first condition). Note that from 
our assumptions on A it follows that such a box does exist. Then we have 
{wi+n+h—i)+{zn—h—n+l) = fli+n—i+Dn—n+l = 0+n— 1)— n+1 = 
0, which means that I\{w{X), z[X)) = 0. Lemma is proved. □ 

Now we can prove the theorem 14.21 First we note that in Okounkov's for- 
mula we can always assume that the diagram A contains the mxn rectangle. 
Indeed, otherwise K^^ a = since in that case one can easily see that 

Cl{d) _ 

qid) 

Therefore by lemma 1^31 

m n 

I-^{w{X),z{X), -1, h) = II J](A,+A;-z-i+l)(Ai-A;.+j-i+2/i-l)x 

i=ij=i 

If,{n, -1, h + n)Ii^{v, -1, 1 - /i + m), 
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where fl, v are defined by A as in (|28|) . We rewrite now the coefficient 
in terms of the diagrams fj, and i^. We have 

m n 

(1) = n n(^« + a;- - i - i + i)c^-(i)c,-;(i), 
i=ij=i 

m n 

C+{2h-l) = \[\[{Xi-X'^+j-i+2h-l)C;{2{h+n)-l)C+{2{h-m)-l), 

i=ij=i 

Clim-n) _ Cl{m)Cl,{-n) 
C9(m-n) C?(m) C?, (-n)' 

CO(m-n-p-<?-i) Cl{m-p-q-\)Cl,{-n-p-q-\) 



C'^{m - n- p-q-\) C^{fn - p- C^/i-n - P - Q - ^) 

Now we use the Berele-Regev factorisation formula [15] for super Schur 
polynomials 

m n 

sp-,{u, V, -1) = (-1)1^1 n n(^^ - ^j)m^)Pi>{v), 

i=ij=i 

where Px{u) are usual Schur polynomials. Comparing ([22|) and ([27|) and 
using Okounkov's formula for usual Jacobi polynomials [9] we see that in 
order to prove the theorem we need to show that 

J„{vi, ...,Vn, -1, -p, -1-q) = (-1)'''' y2Du,i>Pu{v), 

uCi/ 

where 

D - llH~l'>l ^° (-^) ^° ("^ - P - g - ^) -1,1 -h + m) 

C'A-n) Cl,{-n -p-q-\) C;;,{l)C^,{2{h - m) - 1) ■ 

Since 

Clix) = (-1)I-IC70 C-{x) = C;,i-x), = i-l)Wc+i-x), 

we have 

r-nM n = iM-\^\ + P + g + l) -1,1 -h + m) 

^ ' C?(n)C?(n + p + g + i)a-(l)C+(l-2(/i-m)) 

and the proof now follows from Okounkov's formula. □ 

Let's come to the proof of the main theorem 14. li We need the following 
result. Denote the right hand side of the formula (j20p as RHS. 

Proposition 4.4. The right hand side of the formula 120\) can be written 
in terms of super Schur functions as 

RHS = ^ C^{d,p,q)SP^{ui - 2, ... ,Um - 2,vi - 2, ... ,Vn - 2), 
where the coefficients C^{d,p,q) are some rational functions of iJ,,d,p,q. 
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To prove this let us denote r = i(A), s = j(A) and expand every polyno- 
mial , 1 < i < r in terms of the powers of Ui — 2 and every polynomial 
Skjj 1 ^ J ^ s in terms of the powers of Vj — 2. Then RHS is the sum of 
terms 



weSmXSn 



^ . Jui- 2)'i ...{ur- 2y^-{vi - ...{vr- 2)'^ F^Gx 

V) 

A{u)A{v) 

with some constant factors depending on d, p, q, where 

Fx = {Ur+l - 2r'''^ . . . - 2f, Gx = {Vs+1 " 2)™"^-! ...{Vn- 2)° 

and T = (li, . . . ,lr,ki, . . . kg) satisfy the conditions 0<Zj</j,l<i< i(A) 
and 0<kj<kj,l<j< j(A). Since ttx is symmetric with respect to 
ui, . . . ,Ur and vi, . . . ,Vs we may assume that both li and kj are pairwise 
different. Now the proposition follows from theorem 12.11 because of the 
following 

Lemma 4.5. Let ai > ■ ■ ■ > an sequence of nonnegative integers and 
bi, . . . ,bn sequence of pairwise different nonnegative integers such that > 
bi,i = 1, . . . ,n. Let us reorder sequence {bi} in decreasing order b[ > b'2 > 
■ ■ ■ > b'^. Then for any 1 < i < n we have b[ < ai. 

Proof. We will prove lemma by induction on n. The case n = 1 is obvi- 
ous. Assume that lemma is true for some n and consider the sequences 
ai > • • • > a„ > On+i, bi, . . . ,bn, bn+i, which satisfy the conditions of the 
lemma and the corresponding sequence b[ > b2 > • • • > b'^ > b'^_^_l. Let 
c = min{6i, . . . , 6^+1}- If c = We can apply inductive assumption to 

ai, . . . ,an, bi, . . . ,bn. li c = bi bn+i then we can apply inductive assump- 
tion to oi, . . . , an and 61, ... , bi-i, bn+i, ^i+i, • • • , ^n- Lemma is proved. □ 

Let us finish the proof of the theorem 14.11 The proof is by induction on 
m + n. First note that if A contains the m x n rectangle then the theorem 
follows from Theorem 14 . 2 1 and Proposition 7.1 from Okounkov-Olshanski [9]. 
In particular, this is always true when either m or n is zero. 

Assume now that both m and n are positive and A does not contain the 
m X n rectangle. Make the substitution = Vn = t in both sides of 
the formula (|20p . The result is independent on t and reduces to the case 
with smaller number of variables ui, . . . , Um-i and f 1, . . . Vn-i'- for the left 
hand side this follows from Okounkov's formula, for the right hand side from 
proposition 14.41 Thus by induction we have that the difference between the 
right hand side and left hand side of the formula ()20p is divisible by the 
product YYiLi YVj=ii'^i Note that both sides are linear combinations of 

the super Schur polynomials SP^ with A C A, which follows from Okounkov's 
formula and proposition l4.4[ However the ideal generated by HI^i 11^=1 ~ 
Vj) is known to be linearly spanned by the super Schur polynomials <SP^ with 
H containing the mxn rectangle. This means that the difference is actually 
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zero since by assumption A does not contain it. This completes the proof of 
Theorem liH 

As a corollary we have one of our main results. 

Theorem 4.6. The limit of the super Jacobi polynomials SJx{u, v, —l,p, q) 
as {p,q) ( — 1,0) is well defined and coincides up to a constant factor with 
the Euler supercharacter for the Lie superalgehra 05p(2m + 1, 2n).: 

SJxiu, V- -1, -1,0) = 2*W-™i?A(n, v). (29) 

The proof follows from comparison of formulas (jlSp and (j20p . 



5. Fieri formula 

One of the problems with the special value k = —1 is that the spectrum 
of the corresponding ring of quantum integrals is not simple. This means 
that we need more information to characterize the specialized super Jacobi 
polynomials in this case. In this section we derive the Fieri formula for 
super Jacobi polynomials in the case when k = —1 and show that it allows 
to characterize them uniquely. 

This formula can be deduced from the Fieri formula [3j for general k by 
taking the limit /c — > — 1. However the corresponding calculations are quite 
long, so we will do this in a different way. 

Let us introduce some notations. Let 'Pm,n be the set of all partitions A 
from the fat {m,n) hook H{m,n). Let us call a box □ = (i,j) of Young 
diagram A special if 

i- j = d, 

where d = m — n as before is super dimension. We will write /i ~ A if Young 
diagram fi can obtained from A by removing or adding one box. 
Introduce the following functions: 

, j 0, ;u = A \ □ for special □ 

ad(^,A) = < . (30) 

1, otherwise. 



— 1, if A has a removable special box 

1, if there is a special box which can be added to A (31) 
0, otherwise. 



Theorem 5.1. Let SJx{u,v; —1, —1,0) = SJx{u,v) be specialised super Ja- 
cobi polynomials, then the following Fieri formula holds 

m n \ 

^ - ^ + 1 SJxiu, v) = 

,^=l 3=1 j 

^ adii^,X)SJf^iu,v) + bd{X)SJx{u,v) (32) 
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Proof. We need the following Fieri formula for the Jacobi symmetric func- 
tions J'x{u,k,p,q,h) G A, where A is the algebra of symmetric functions 
|18j . h is an additional parameter (see [3]). First note that the limit of 
J'x(u,k,p,q,h) when k ^ —1 for generic p,q is well defined as it follows 
from Okounkov's formula. We denote this limit J^x- By Aztsj we denote the 
sets (Ai, . . . , Aj_i, Aj ± 1, Aj+i, . . . ) respectively. 

Lemma 5.2. For k = —1 and generic p,q the Jacobi symmetric functions 
satisfy the following Fieri formula 

PiJx = ^^+^' + "(^^ + d-i)\jx (33) 

where pi = ui -\- U2 -\- - . . ^ d = h -{- + q and 

2p{p + 2q + l) 

{2l-p-2q-l){2l-p-2q + l)' ^'^^> 

^ 2l{2l -2q- 1)(2/ -2p-2q- 1){21 - 2p - Aq - 2) 

{2l-p-2q){2l-p-2q-lY{2l-p-2q-2) ' ^ ' 

To prove the lemma we use the following formula for the Jacobi polynomi- 
als with k = —1 in m variables (see Froposition 7.1 in Okounkov-Olshanski 

Jx{u,-l,p,q) = ^ e{w)w[fx-,+m-l{ui)fx2+m-2{u2) ■ ■ ■ fx^iUm)] , 

where as before fi{z) = fi{z,p,q) are the classical normalized Jacobi poly- 
nomials in one variable with parameters p, q. They satisfy the following 
three-term recurrence relation (see e.g. |21]): 

zfiiz) = fi+,{z) + a{l)fi{z) + b{l)fi_,{z) 

with a{l), b{l) given by ()34p . (l35p . Therefore we have 

^Ui Jx{u) = — — Y [u^fx,+nl--l{ul) . . . fx^{Urn)] 

1=1/ ^ ' i=l weSra 

/m m \ 

= Y '^A+£,(^^)+^a(Ai + m-z)+ Y a{m-i)\jx{u) 

i:X+ereVm,n \i=l i=l{X)+l J 



+ Y K^i + m-i)Jx-e,{u), 



r-X—ei&Vm,!!, 
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where Jx{u) = Jx{u, —l,p, q) and /(A) is the number of non-zero parts in 
partition A. Since 

pjp + 2q + l) pjp + 2g + 1) 

a{x) = 

^ ^ 2x-p-2q + l 2x-p-2q-l 

we have 

p{p + 2q + l) 



a{m — i) = p + 



.^KAm 2m-2l{X)-p-2q-l 



Comparing this with ([33]) we see that this formula is true after the natural 
homomorphism A A^, A^ is the algebra of symmetric polynomials of m 
variables, if we specialize d to m. Since this is valid for all m, the lemma 
follows. 

The super Jacobi polynomials SJx{u, v, —l,p, q) are defined as the image 
of Jacobi symmetric functions J7a under the homomorphism 

m n 

i=l i=l 

where d is specialized to m — n and Am,n is the algebra of supersymmetric 
polynomials (see e.g. [18]). Computing the limits when {p,q) (—1,0) 

lim ail) = 6(1 + 1) - 6(1), lim b(l) = 1 - 6(1) 

(P,<?)^{-1,0) (p,<y)^(_i,o) 

lim (p + , + + ) = -1 + 5id - /(A)), 

(p,g)^(_i,0) V 2d-2l{\) -p-2q-l) ^ ^ 

we have the following formula 

m n \ 

- ^Vj SJx{u,v) = 

,i=l 3=1 I 

SJx+eAu,v)+ Yl [l-S{K-i + d)]SJx-eSu,v) (36) 

/(A) 

+ Y ['^(^* -i + d+l)- 6{\, -i + d)] SJxiu, v)+[6id - /(A)) - 1] SJxiu, v), 
1=1 

where d = m — n and 

'l, x = 
0, X / 0. 

One can check that it is equivalent to the Fieri formula (|32p . □ 



6{x) 



Remark 5.3. The form of the factor on the left hand side of Fieri formula 
\3S^) . which was chosen by convenience, has a clear representation-theoretic 
meaning: Yl^i ~ Sj=i + 1 ^•s supercharacter of the standard rep- 
resentation of 05p(2m + 1, 2n). 
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Remark 5.4. One can possibly use this for the alternative proof of the 
main theorem. For this one should only prove that the Euler supercharacters 
satisfy the corresponding version of the Fieri formula. This is related to the 
translation functors in representation theory (see e.g. [20\). 

6. Jacobi-Trudy formula for Euler supercharacters 

In this section we give one more formula for specialized super Jacobi 
polynomials (and hence for the Euler supercharacters) of Jacobi-Trudy type. 

We start with the Jacobi-Trudy formula for Jacobi symmetric functions 
following [22]. Let J^x €z A he the Jacobi symmetric functions with k = —1 
and generic p and q (see [3j). They depend also on the additional parameter 
d replacing the dimension of the space. Let hi = Jx for the partition A = (i) 
consisting of one part for positive i and /ij = if i < 0. Define recursively 
the sequence /i- € A, i € Z by the relation 



h 



(r+l) 



+ a(i + d - ^ + 6(i + d - \)h 



(r) 



At) 



(37) 



for r = 0, 1, 



h{x) 



. with initial data h, 
a{x) = 
2x{2x 



(0) 



hi and 



2p{p + 2g + 1) 



{2x-p-2q- l)(2x -p-2q + l)' 
2q - l)(2x -2p-2q- l){2x - 2p - Aq 



2) 



{2x-p- 2q){2x - p - 2q - iy{2x - p - 2q - 2) 

Theorem 6.1. [22] The Jacobi symmetric functions with k = —1 have the 
following Jacobi-Trudy representation: 

hxi 
hx2-i 



Jx 



where I = 1{X). 

Taking the limit p — 
we have the following 



Ai 

A2-I 



hxi-i+1 



^A,-Z+l 



(38) 



-1, g — > and the homomorphism • 



Corollary 6.2. The specialized super Jacobi polynomials SJx satisfy the 
following Jacobi-Trudy formula 



SJx{u,v; -1, -1,0) 



hxi 
/1A2-1 



(1) 

Ai 



h 



hf-'^ 
Ai 



h 



Xi-l+1 "'Xi-l+1 



(1) 



(i-i) 

Xi-l+l 



(39) 



where A G H„ 



I, d = m — n and h. 
a{x) = 5{x + 1 



(r) 



are defined recursively by ([5 
1 - 5{x) 



with 



(5(x), h{x 

18 



and h\ = hi = SJ\{u,v]— 1,-1,^) for A = (i) for positive i and h\ = 
hi = if i < 0. 



7. EULER SUPERCHARACTERS FOR DIFFERENT CHOICE OF BOREL 

SUBALGEBRA 

It turns out that the relation with super Jacobi polynomials can be made 
more direct if we choose, following to Gruson and Serganova [10], a different 
Borel subalgebra and suitable parabolic subalgebras. 

We consider again the case g = 05p(2m + 1, 2n), assuming for convenience 
at the beginning that m > n. Choose the following set of simple roots 

B = {ei — £2, • • • , Em-n+l —^1,^1 — ^m-n+2, £m-n+2 — ^2, ■ ■ ■ ,£m — ^n, ^n} 

This choice is special since this set contains the maximal possible number 
of isotropic roots. The corresponding set of even positive roots is 

= {si ±ej, i < j, 6p ±6q, p< q, 25p} 

with the half-sum 

2^ ^ ]^ 3 1 

Po = 2 a = (m.--)ei + (m--)e2 \- -em + n5i + {n-l)62-\ l-Sn- 

The set of positive odd roots is 

Rf = {si ± 6j, i — j < m — n, 6j ± Si, i — j > m — n, 5j, j = 1, . . . ,n} 
with the half-sum 

pi = n(eiH \- Em-n+i) + (n - l)em-n+2 ^ \-£m. + {n-^)5i-\ '^^^n, 

so 

m—n ^ ^ m 1 " 

P = Po-pi= ^{m-n-i + -)£i - - ^ + 2 '^J- 

1=1 j=m— n+l j=l 

The following lemma gives a description of the highest weights with re- 
spect to our choice of simple roots B in terms of partitions. We need this 
to establish the relation with super Jacobi polynomials. 

Lemma 7.1. The weight 

X = aiEi + 0262 H h am£m + blSl + 62(52 H h bn6n 

is a highest weight of an irreducible finite dimensional osp(2m -|- l,2n)- 

module if and only if there exists a partition A E H{m, n) from the fat 
hook such that 

i=i i>i{x) j=i j>jW 
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The proof is a geometric reformulation of the conditions on the highest 
weights given in [10], Corollary 3. 

Let A be a partition from H{m, n) and x be the corresponding highest 
weight. Consider the maximal parabolic subalgebra p = p(A) such that x 
can be extended to p as a one dimensional representation. One can check 
that in that case the corresponding roots are 

Rp = {±ei ± Ej, ±ei, ±6p ± 6g, ±6p, ±25p, ±ei ± 6p} 

with i(A) < i, j < m, j, j{X) < p,q < n, p ^ q. They correspond to 
the subalgebra osp(2/ + 1, 2/), I = m - i(A) = n - j(A). 

Let E^{x) be the corresponding Euler supercharacter given by (jlip . Let 
TTx be the same as in section 2 and define t{X) as the number of pairs £ 
-7r(A) with i — j>d = m — n. Similarly let s(A) be the number of pairs 
S A with i — j > d. 

Theorem 7.2. Letp andx be the parabolic subalgebra and its one- dimensional 
representation determined by a partition A G H{m,n), then the specialized 
Jacobi polynomial SPx{u,v, —1, —1,0) coincides up to a sign with the cor- 
responding Euler supercharacter: 

SPxiu, V, -1,-1,0) = {-l)<'^E^{x). (41) 
Proof. According to general formula (111(1 

- e"°/^) e^^P 




where 



1 _ 1 

3 ^3 ' 



T = - a. 
2 ^ 

A simple calculation shows that 

n (e"/^-e-«/^)=(-i)*(^) n (-^--.•) n (y. 

a(^RpnR+ {i,j)&7Tx 3<3W 

where Ui = Xi -\- x^^ , Xi = e^S i = 1, . . . ,m, Vj = yj -\- yj^, yj = e^^ , j 
1, . . . ,n. One can check also that 

m J i(A) ^ n 

X + P + r = + -)ei + + -)5j + (n - j + l)5j, 

i=l j=l j>j{X) 

where U, kj are defined by (I15p . ()16p . Therefore we have 

EP{x) = (-i)*w T w( - ^^■) n,<,(A)^(4 - ^7^)^^ +"+^ 
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w^Wo 



n 



V 



A{u)A{v)ll( 



) n (^1+^7') n (y.-y.") 



i<i(A) 



i>i(A) 



Now we use the identities 

yi 



yi 2/2 



= 1= E ^ 

where Wi is the Weyl group of type Ci « BCi, which is a semi-direct prod- 
uct of permutation group 5/ and Zg. These identities foUow from the Weyl 
(super) character formula for trivial representations of sp(2/) and osp(l,2/). 
This leads to 



w£Wo 



n iui-vj)xi 



■ ■ ■ J-m yi 



■■■Vn 



\ 



A(n)A(^)n( 



-xr')\{{y] +yj') 
i=l j=l 

Comparing this with Theorem 14.11 and using the obvious relation 



s{X)=tiX) + b{\), b{X) = J2^i 



we have the claim. 



□ 



8. The case of osp(2m,2n) 



In this section we present the results in the even orthosymplectic case 
g = osp(2m, 2n). 

It would be instructive to start with the special case n = 0, i.e. with the 
usual orthogonal Lie algebra o(2m). It has the root system of type with 
simple roots 

We have a symmetry Em —£m, corresponding to the (outer) automor- 
phism 9 of this Lie algebra. This automorphism appears in the description 
of the representations of the corresponding Lie group 0{2m), which consists 
of two connected components. 

Recall (see [6]) that the highest weights fj, = fj-iEi + • • • + fJ-m^m of ir- 
reducible finite-dimensional representations of Lie algebra o(2m) have the 
following form: all fii are either integer or half-integer and satisfy the in- 
equalities: 

IJ-l > fJ'2 > ■ ■ ■ > fJ'm-l > |/^m|- 

The half-integer fi correspond to the spinor representations and can not 
be extended to the representations of the orthogonal group SO {2m), so we 
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restrict ourselves by integer //. Corresponding representation of o(2m) 
can be extended to the full orthogonal group 0(2m) if and only if it is 
invariant under the automorphism 9, which is equivalent to /x^ = 0. If 
A*m 7^ then one should consider the direct sum 

which gives an irreducible representation of 0{2m). 

It is interesting that the sum of the corresponding Euler supercharacters 
appears also in the general orthosymplectic case q = osp(2m, 2n) as the limit 
of super Jacobi polynomials (see below), so these limits are natural to link 
with supergroup OSP{2m,2n) rather than Lie superalgebra osp(2m, 2n). 

Now let us give precise formulation of the results. We have = 0o © 
01, where 0o = so(2?ti) © 5p(2n) and 0i = Vi ^ V2 where Vi and V2 
are the identical representations of so(2?tt-) and 5p(2n) respectively. Let 
ibei, . . . , ±em, ±(^1, • • • , ±<5n be the non-zero weights of the identical repre- 
sentation of 0. The root system of osp(2m, 2n) consists of 

Ro = {±ei ± Ej, ±6p ± 6g, ±26p, i ^ j, I < i,j < m ,p q, 1 < p,q < n}, 

Rl = Riso = {±^1 ± Sp}, 

where Ro,Ri and Riso are even, odd and isotropic parts respectively. The 
Weyl group Wq = (^Sm x ^2'""^''^ x (5„ k Zg) acts on the weights by sep- 
arately permuting Ej, j = 1, . . . , m and 6p, p = 1, . . . ,n and changing their 
signs such that the total number of signs of is even. A distinguished 
system of simple roots can be chosen as 

B = {Si — 62, ■ ■ ■ , 6n-l — Sn, (5„ — ei, ei — £2, • • • , Em-l — Em-l + ^m}- 

We have again a symmetry — > —Em, corresponding to the automorphism 
of Lie superalgebra osp(2m, 2n), which also denote 9. It acts also in a natural 
way on the Grothendieck ring and supercharacters. 
Consider the parabolic subalgebra p with 

= {^i - £j, Sp - Sq, ±{£i - 6p)}, 

which is isomorphic to the sum of the Lie superalgebra 0[(m, n) and some 
nilpotent Lie superalgebra. For any finite-dimensional representation M of 
p the corresponding Euler supercharacter E^{M) is given by (jlip . which in 
this case has the form 



^P(M) = J2 w 



wGWo 



/ n (1 - ^^%')^T'' ■ ■ ■ xlVl ■ ■ ■ ynSchM\ 

A{u)A{v)UU^y,-yj') 
V / 



(42) 

Here Ilm,n is the rectangle of the size mx n, A{u) = YYi<ji''^i ~ ""j)? A.{v) = 
Ili<ji'^i ~ ^i) Ui = Xi + x~^, Xi = Vj = Hj + yj = e^^ as before. 
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Proposition 8.1. For the trivial even representation M we have 

SP(M) = 2™''"(""-^'"). (43) 
Proof. The proof is similar to Proposition 3.1 and is based on the formula 

where 5a is the Schur polynomial and the sum is over all the Young diagrams 
A, which are contained in the (m x n) rectangle. Replacing in the formula 
(f42ll as before the product Yii ^(1 — x~ yj ) by the sum 

.n 

one can check that 

• • • -^m ill ■ ■ ■ yn I _ „ 

unless A is either empty or partition of the form 

A = (ai, . . . , a^lai + 1, . . . , Or + 1) 

in Frobenius notations (see e.g. [18]), in which case it is equal to (— l)''^'. 
Now the proposition follows from the fact that the number of such diagrams 
contained in the (m x n) rectangle is equal to 2™™*^™"^'"), which can be 
proved by induction or reduced to the odd case. □ 

Let M = be the polynomial representation of g[(m,n) determined 
by a Young diagram A G H{m, n) and define the Euler supercharacters as 
E\ = E{V^). Introduce the following polynomials 

^a{z) =w'' + W ''^ a>Q, ^Q = l, ^IJa{z) = , (44) 

vo — w ^ 

where z = w + w~^. They are particular case of the Jacobi polynomials 
known as Chebyshev polynomials of the first and second kind respectively. 

Let i{X),j{\),ki,lj be defined by as before by (II]),(l2|),([l5D,([l6]). Define 
the following modification of i(A): 

i*(A) = max{i |Aj + d — i>0, l<i< m}. 

It is clear that that i*{X) is equal to i(A) — 1 or to i(A) depending whether 
h{X) = or not. 

Similarly to Theorem 3.2 one can prove that 

Theorem 8.2. If Im = then the Euler supercharacters for 05p(2m,2n) 
can be given by the following formula 



E^ = C{X) Yl ^ 

weSmXSn 



Ilx{u,v) 



A{u)A{v) 

(45) 
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where C{\) = {-ifT^-^'W--^, b = Ej>m^j- 

If lm> ^ then we have a similar formula for the sum of Euler superchar- 
acters 

(fl^ (ui)... <fi,„ (vi)-.. ipk„ (Vn) ' 



W&SmXSn 



'nx{u,v) 



A{u)A{v) 



(46) 

As a corollary we have 

Theorem 8.3. The limit of the super Jacobi polynomials SJ\{u, v, —l,p, q) 
as (p, q) — > (0, 0) is well defined and coincides up to a constant factor with 
the Euler supercharacter or sum of two Euler supercharacters: 

S.h{u,v--IM = 2**(")-'"+^i^A(n,7;) (47) 

if Im = 0, and 

SJx{u,v;-l,0,0)=Ex + e{Ex) (48) 

iflm > 0. 

The proof follows from comparison of formulas (j45p , (j46p with ()20p . 

The Fieri formula for the Euler supercharacters of 05p(2m, 2n) follows 
from Fieri formula for super Jacobi polynomials with p = q = 0. Introduce 
the following functions: 

0, ;U = A \ □ and j — i = —d 

adifi, A) = <2, fi = X\n and j - i = 1 - d (49) 

1, otherwise, 

As before /u ~ A means that the Young diagram n can obtained from A by 
removing or adding one box. 

Theorem 8.4. Let SJx{u,v; —1,0,0) = SJx{u,v) be specialized super Ja- 
cobi polynomials, then the following Fieri formula holds 



^ adilJ'A)'SJf,{u,v) (50) 



i=i j=i 



/i~A, fJ.€Vm,n 



The Jacobi- Trudy formula in this case follows directly from (|38p . 

Proposition 8.5. The specialized super Jacobi polynomials satisfy the fol- 
lowing Jacobi-Trudy formula 



SJx{u,v; -1,0,0) 



hx, 



h 



(1) 



h 



Ai 

(1) 



A2-I 



hf-'^ 
Al 

'''X2-I 



K 



(i-i) 

Xi-l+l 



(51) 



where A G Hm,n, d = m — n and hi are defined recursively by p7\ ) with 
a{x) = 0, b{x) = 1 + 5{x - 1) - 5{x) 
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and hi = hi = SJx{u,v; —1,0,0) for A = (i) for positive i and h\ — hi = 
ifi<0. 

Consider now a different choice of Borel subalgebra and suitable parabolic 
subalgebras, following Gruson and Serganova |10j . 

We assume for convenience that m > n. Choose the set of simple roots 
with maximal number of isotropic roots: 

B = {Si —82,..., Sra-n " (^l, (^1 — Em-n+li ^m-n+l — ^2., ■ ■ ■ , — ^m, + ^m} . 

The corresponding set of even positive roots is 

^o" = i < i> ±^q, P < q, 25p} 

with the half-sum 

po = ^ ^ a = {m-l)ei + {m-2)s2 \-em-i + n6i + {n-l)62-\ h(5„. 

The set of positive odd roots is 

Rf = {si ± 6j, i — j < m — n, 6j ± Si, i — j > m — n} 
with the half-sum 

Pi = n{ei H h ffm-n) + (n - l)em_„+i H h Sm-i + n6i A h 

so 

m—n 

p = Pq- pi= ^ (m - n - i)ei 

i=l 

Lemma 8.6. The weight 

X = aiEi + 0262 H h arnEm + ^l^l + ^2^2 H h ftn^n 

with am > is a highest weight of an irreducible finite dimensional 05p(2m, 2n)- 
module if and only if there exists a partition A G H{m, n) from the fat hook 
such that 

i(A) j(A) 
X + P = J^(A^ + d- i)e, + ^(A;. -d-j + l)6j (52) 
i=i j=i 

The proof again follows from comparison with Corollary 3 from [TO] . 
Let A be a partition from H[m, n) and x be the corresponding highest 
weight. Consider the parabolic subalgebra p = p(A) with 

Rp = {±ei ± Ej, ±5p ± 5q, ±25p, ±Ei ± 5p} 

with i{X) < i, j < m, i ^ j, < p, q < n, p q , corresponding to the 

subalgebra osp(2/, 21), I = m — i{X) = n — j(A). 

Let E^{x) be the corresponding Euler super character given by (fTT|) . Let 
TTx be the same as in section 2 and define t(A) as the number of pairs {i,j) € 
7r(A) with i — j>d = m — n. Similarly let s(A) be the number of pairs 
{i,j) S A with i — j > d. 
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Theorem 8.7. Letp andx be the parabolic subalgebra and its one- dimensional 
representation determined by a partition A € H{m,n), then 

SJx{u, v; -1, 0, 0) = (-l)^W2** W-^WeP(x) (53) 

if Am < n, and 

SJxiu,v; -1,0,0) = i-lf^\E^ix) + eiE^ix))) (54) 

if Am > n. 

Proof. According to general formula (llip 



where 

r = ^ ^ a=^(m- i)ei + ^ (n - j + l)(5j. 
One can check that 

n (e"/^-e-"/^)=(-i)*w n (^^--.) 

and that 

m n 

X + p + T = ^i^i + ^i^j + 

i=l i=l 

where li, kj are defined by (jlSp . p6p . Therefore we have 

nA(u,v)e^+^+^ \ 



£;P(;t) = (-1)*W y w; ( - „ , , - 

(_,.t(x) V- / ^A(^.,^;)x^..x:;^y^+^..^/^+l ^ 

^ ^ ^ ^\ A(u)A(v) i ■ 

If /m = then it is easy to see that the average over Z™^^ x gives 

TPPf \ n(\\ f Uxiu,v)Lpi^ (Ui) . . .(/?;^(Um)V^fci+l(^l) • ■■A„+l{Vn) \ 

with D{X) = (-l)*W2^W-**W. If /m > then we should consider the sum 

E^ix) + eiE^ix) = 

/_.y(A) C^xju, V) (Ph jui) • • • '^lmiUm)i'ki+livi) ■ ■ ■ '4'k„+liVn) \ 

^ ^ .^.h. V A(^)A(.) 

Theorem 14.11 and tl 
t(A) = s(A) + 6(A). 



WGSmXSn 

Now the claim follows from Theorem 14.11 and the relation 



□ 
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Remark 8.8. The coefficient 2*(-^)-**(^) (which can be either 1 or 2) can he 
eliminated by a different choice of parabolic subalgebra. Indeed, in the case 
when i*(A) = i(A) — 1 our choice of parabolic subalgebra p is not the maximal 
one, which in that case corresponds to osp(2/ + 2,2/). 

9. Concluding remarks 

We have shown that Euler supercharacters for orthosymplectic Lie super- 
algebras coincide with speciaUzed super Jacobi polynomials. This fact seems 
to be very important and should have more conceptual proof. One possibil- 
ity is to show that the Euler supercharacters can be uniquely characterized 
by Fieri formula and the fact that they are the common eigenfunctions of 
the corresponding algebra of quantum integrals of deformed Calogero-Moser 
problem. From representation theory point of view this gives the action of 
the translation functors on Euler supercharacters. The results of Brundan 
|20 ] show that one might be able to characterize in a similar way the irre- 
ducible characters. 

Another interesting question is about geometry of the genuine parameter 
space of the (super) Jacobi polynomials. We have seen that the specializa- 
tion process requires some blowing-up procedure. The calculations in the 
special case of 05p(3,2) indicate that this may lead to a description of the 
characters of important classes of finite-dimensional representations. 
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